Puncturing the Propaganda Bubble:
An Experiment on Attention to News in Russia

Georgiy Syunyaev
g.syunyaev@vanderbilt.edu

Vanderbilt University, WZB Berlin

Anton Shirikov
shirikov@ku.edu

University of Kansas

Motivation

  • Most people consume news from outlets that are biased

    • This can mean misinformation, framing, or selection
    • More likely in autocracies with state control of the media environment

  • But often they misunderstand or underestimate media bias

  • Can we correct these misperceptions and what are the downstream effects?

Question: Does highlighting media reporting patterns…

…lead to learning about bias, switching to less biased media, and changes in political attitudes?

How can we answer the question

  • Dual-processing theory: Type 1 processing – automaticity and cue, and Type 2 processing – deliberation

  • Media literacy: Activate Type 2 processing via innoculation or prebunking interventions to improve ability to discern biased or false information

    • Can fail or have differential response, e.g. due to motivated reasoning

  • Attentiveness prompts: Activate Type 2 processing by asking citizens about information they consume (e.g. whether it is accurate)

    • But propaganda or biased media does not always lie
  • We combine approaches to see if citizens can learn about pro-government and independent media on their own

Our approach

Ask citizens to analyze the topics and sentiment of news headlines by state and independent media

…to test whether…

Paying attention to what media outlet covers make citizens:

  1. Learn about media reporting patterns (positive domestic and negative foreign)

  2. Change perceptions media outlet’s bias (state is more biased)

  3. Adjust media consumption (away from state media)

  4. Change political attitudes (more critical of government / policies)

Setting and design

  • Russia: an autocratic regime with state control of media environment

    • The full-scale invasion in Ukraine exacerbated state media control
    • State media is heavily pro-government (e.g., Rossiya-1)
    • Some more critical media is still available (e.g., RTVI)

    \(\Rightarrow\) Harder to shift media beliefs and consumption

  • Study:

    • 4-wave online panel experiment among 1176 adult Russian citizens enrolled in online panel
    • Simple random assignment to one of three experimental groups with different sets of tasks
    • Content-analysis tasks at the end of survey in waves 1-3 \(\Rightarrow\) more than 2 weeks between tasks and outcome measurement
    • Analyses focus on the endline outcomes

Intervention

  • Watch 4 to 6 short news headline segments per wave from one of three TV channels:

    • Kultura, a non-political TV channel that covers art, theater, architecture, etc.
    • Rossiya-1, the main state-controlled propaganda channel
    • RTVI, a privately owned TV channel with moderately critical editorial policy

    \(\Rightarrow\) In our sample 67% watch Rossiya-1; 3% (!) watch RTVI

  • After each video, six questions asking to count frequency of discussion of:

    • Situation in Russia in a [negative/positive] tone?
    • Russian federal officials in a [negative/positive] tone?
    • Situation in other countries in a [negative/positive] tone?

Experimental conditions

  • Placebo: Kultura (Date 1) \(\Rightarrow\) Kultura (Date 2) \(\Rightarrow\) Kultura (Date 3) \(\Rightarrow\) Kultura (Date 4) …
  • State: Rossiya-1 (Date 1) \(\Rightarrow\) Kultura (Date 1) \(\Rightarrow\) Rossiya-1 (Date 2) \(\Rightarrow\) Kultura (Date 2) …
  • State + Independent: Order: Rossiya-1 (Date 1) \(\Rightarrow\) RTVI (Date 1) \(\Rightarrow\) Rossiya-1 (Date 2) \(\Rightarrow\) RTVI (Date 2) …

  • Total of 20 videos to code per respondent over the course of a month

Awareness of media reporting patterns

  • Control group: More uncertainty about independent media coverage
  • Evidence of learning about coverage patterns
  • Especially for independent media in respective group

Perceptions of media outlets

  • Control group: State media is more biased than independent
  • No change in perceptions of state media
  • But improved perceptions of independent media in respective group

Media consumption and preferences

  • Control group: Likely to consume state media
  • Positive shift in self-reported consumption of both Rossiya-1 and RTVI
  • Stronger revealed preference for RTVI in both treatment groups
  • Increased awareness of both pro-government and critical news

Political attitudes

  • Lower support for the government and stronger concerns about Ukraine

  • No clear differences between treatment groups

Mechanisms


  • Changes in perceptions of independent media and interest in it only among Putin critics, not supporters
  • News awareness: pro-gov’t news \(\Uparrow\) mainly among supporters, critical news \(\Uparrow\) mainly among critics
  • Approval of authorities decreased, and concern about Ukraine increased among Putin critics and independent media consumers

Interpretation: Attention to news can leads to interest in new (more balanced) media and change in political attitudes when it aligns with prior dispositions/media diets; otherwise, little change occurs.

Takeaways


  • Simply asking citizens to pay attention to what is covered in the news (pro-government/independent) can change political attitudes
  • This does not happen because of learning about bias or shifting away from the popular propaganda
  • Instead plausible channel is interest in new, more balanced, media sources as comparison and changes in how the media is consumed
  • The political effects are limited to those who are more critical of the government in the first place

Appendix

References

Sample characteristics

Variable Obs Mean SD Min p25 p50 p75 Max
Socio-economic characteristics
Age 1176 44.81 12.06 19 36.00 45.00 54.00 80.00
Female 1175 0.48 0.50 0 0.00 0.00 1.00 1.00
Education 1176 2.55 0.68 0 2.00 3.00 3.00 3.00
Income level (categorical) 1176 2.82 0.88 0 2.00 3.00 3.00 5.00
Employed permanently 1176 0.74 0.44 0 0.00 1.00 1.00 1.00
Use of advanced Internet technologies
VPN 1050 0.50 0.50 0 0.00 1.00 1.00 1.00
Total (out of 8) 1176 3.22 2.13 0 2.00 3.00 5.00 8.00
Overall news consumption
Frequency (categorical) 1176 2.74 0.63 0 3.00 3.00 3.00 3.00
From TV 1142 0.80 0.40 0 1.00 1.00 1.00 1.00
From online media 1142 0.73 0.44 0 0.00 1.00 1.00 1.00
Recently consumed news
From Rossiya-1 1175 0.68 0.47 0 0.00 1.00 1.00 1.00
From RTVI 1175 0.04 0.19 0 0.00 0.00 0.00 1.00
Intensity of state media use 1173 0.24 0.18 0 0.12 0.19 0.31 0.88
Intensity of indep. media use 1173 0.07 0.15 0 0.00 0.00 0.11 1.00
Attitudes
Media in Russia unbiased 1176 0.52 0.28 0 0.33 0.67 0.67 1.00
Strength of preference for unbiased news 1176 0.36 0.23 0 0.25 0.25 0.50 0.75
President approval 1173 0.60 0.49 0 0.00 1.00 1.00 1.00
Government focuses too much on
Law and order 1176 0.09 0.29 0 0.00 0.00 0.00 1.00
Foreign affairs 1176 0.12 0.33 0 0.00 0.00 0.00 1.00

Concerns over the design?


  • No pure control? No systematic evidence of Placebo (Kultura) group changes over time

  • Political news push respondents away? No differences in patterns of attrition across treatment groups

  • Less attention to political news? High and similar rates of correct answers about the topic and channel across groups

  • Experimenter demand? No differences across treatment groups in respondents’ assessments of study goals

Media perceptions breakdown

Authorities approval breakdown

Policy satisfaction breakdown